Skip to content

Optimize grant macros for performance and case-insensitive role matching#33

Merged
jonhopper-dataengineers merged 2 commits into
mainfrom
feature/grant-object-remove-revokes
Jun 25, 2026
Merged

Optimize grant macros for performance and case-insensitive role matching#33
jonhopper-dataengineers merged 2 commits into
mainfrom
feature/grant-object-remove-revokes

Conversation

@jonhopper-dataengineers

@jonhopper-dataengineers jonhopper-dataengineers commented Jun 25, 2026

Copy link
Copy Markdown
Member

Summary

  • Performance optimization: All grant macros now check existing state before issuing SQL statements. Schemas/objects where grants are already in place are skipped entirely, reducing runtime from 20-30 minutes to under 2-3 minutes on large databases.
  • Case-insensitive role matching: All role/application name comparisons are now case-insensitive via uppercase normalization. Passing ['analyst'], ['Analyst'], or ['ANALYST'] all work correctly.
  • Grant-only mode for grant_object: The grant_object macro no longer revokes privileges — it only ensures specified privileges are granted to specified roles.
  • New helper macros: Added _grants_get_schema_object_privs, _grants_get_schema_grants, _grants_get_future_grants, and _grants_normalize_roles to _helpers.sql.
  • Integration tests: Added est_grants_helpers (13 pure Jinja unit tests) and est_grants_idempotency (9 integration tests verifying skip-logic and helpers against live Snowflake).
  • Version bump: 1.0.6 -> 1.0.7

Test plan

  • Run dbt run-operation test_grants_helpers — validates helper macro logic (no Snowflake connection needed)
  • Run dbt run-operation test_grants_idempotency — validates helpers return correct types and skip-logic works against live data
  • Run dbt run-operation grants_smoke_test --vars '{"grants_dry_run": true}' — exercises all major grant macros in dry-run mode
  • Run dbt run-operation grant_privileges on a test environment and verify reduced statement count in logs
  • Run grant_privileges twice in succession and confirm second run logs "no changes required" / "schemas skipped"

.... Generated with Cortex Code

Summary by Sourcery

Optimize Snowflake grant macros for faster, idempotent execution and case-insensitive role handling while making grant_object grant-only.

Enhancements:

  • Add helper macros for bulk inspection of schema-, object-, and future-grant state and reuse them across grant workflows.
  • Refine grant macros (schema read/monitor/operate, procedure usage, share read, database usage, application grants) to normalize role/application names and avoid issuing redundant grant statements.
  • Update grants_smoke_test to validate case-insensitive role behavior and broaden coverage of core grant macros.
  • Adjust logging in grant macros to report executed statement counts and skipped schemas for clearer operational insight.

Documentation:

  • Update README and CHANGELOG to document grant-only behavior for grant_object, performance improvements, and version bump to 1.0.7.

Tests:

  • Add Jinja-based helper and integration tests to validate grant helper behavior, case-insensitive role normalization, and idempotent/skip-logic behavior against live Snowflake.

Chores:

  • Bump package version from 1.0.6 to 1.0.7 in project configuration and documentation.

…atching

Grant macros now check existing state before issuing SQL statements, skipping
schemas/objects where grants are already in place. This reduces runtime from
20-30 minutes to under 2-3 minutes on large databases where grants are already
applied. Also makes all role comparisons case-insensitive via normalization to
uppercase. Adds integration tests for grant helpers and idempotency.

.... Generated with [Cortex Code](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code)

Co-Authored-By: Cortex Code <noreply@snowflake.com>
@sourcery-ai

sourcery-ai Bot commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Reviewer's Guide

Optimizes all major grant macros for Snowflake by introducing bulk state-check helpers, case-insensitive role/application normalization, and grant-only behavior, plus new tests and version bump.

Sequence diagram for optimized grant_schema_read_specific flow

sequenceDiagram
    actor DbtRunner
    participant grant_schema_read_specific
    participant GrantsHelpers
    participant Snowflake

    DbtRunner->>grant_schema_read_specific: invoke with schemas, grant_roles
    grant_schema_read_specific->>GrantsHelpers: _grants_normalize_roles(grant_roles)
    grant_schema_read_specific->>GrantsHelpers: _grants_collect_roles()

    loop for each schema
        grant_schema_read_specific->>GrantsHelpers: _grants_get_schema_grants(schema, USAGE, ROLE)
        grant_schema_read_specific->>GrantsHelpers: _grants_get_schema_object_privs(schema, [SELECT,REFERENCES,REBUILD,READ], grant_roles)
        alt include_future_grants
            grant_schema_read_specific->>GrantsHelpers: _grants_get_future_grants(schema)
        end
        alt revoke_current_grants
            grant_schema_read_specific->>Snowflake: run_query(revoke_query)
            Snowflake-->>grant_schema_read_specific: tbl_privs
        end
        alt no missing privileges
            grant_schema_read_specific->>DbtRunner: log("schema skipped, no changes")
        else missing privileges
            grant_schema_read_specific->>Snowflake: run_query(grant statements)
        end
    end

    grant_schema_read_specific->>DbtRunner: log(summary with schemas skipped)
Loading

File-Level Changes

Change Details Files
Grant macros now use bulk privilege/usage/future-grant introspection to only issue SQL statements when privileges are missing, and to skip schemas/objects with no changes required.
  • grant_schema_read_specific now queries information_schema.object_privileges and SHOW FUTURE GRANTS once per schema, builds per-schema statement batches, and logs skipped schemas when everything is already granted.
  • grant_schema_object_privileges replaces per-object SHOW GRANTS loops with a single _grants_get_schema_object_privs call and an information_schema-based revoke query, then issues bulk GRANTs per role/privilege.
  • grant_schema_procedure_usage_specific replaces per-procedure SHOW GRANTS with an information_schema.object_privileges query for USAGE on PROCEDURE, grants usage only for roles missing it, and tracks per-schema skip/execution counts.
  • grant_schema_operate_specific and grant_schema_monitor_specific compute existing OPERATE/MONITOR grants once via information_schema.object_privileges, only grant usage/operate/monitor when missing, and log how many schemas were skipped.
macros/grants/grant_schema_read.sql
macros/grants/grant_schema_object_privileges.sql
macros/grants/grant_procedure_usage.sql
macros/grants/grant_schema_operate.sql
macros/grants/grant_schema_monitor.sql
Share-related grant macros now avoid redundant grants by introspecting existing share state via DESC SHARE and skipping schemas/views that are already granted.
  • grant_external_share_read builds an existing_share_objects list from DESC SHARE, then for each schema only grants usage/tables/views when they are not already present, with per-schema skip/execution logging.
  • grant_internal_share_read mirrors the external-share optimization, using DESC SHARE to determine if schema usage or table grants already exist and only granting missing items, with summary logging.
  • grant_share_read_specific_schema now constructs a per-share map of existing grants from DESC SHARE and only issues usage/select grants for schemas/views not yet in the share, logging when no changes are required.
macros/grants/grant_external_share_read.sql
macros/grants/grant_internal_share_read.sql
macros/grants/grant_share_read.sql
Role and application names are normalized to uppercase across grant macros to ensure case-insensitive matching against Snowflake grantee names.
  • Introduced _grants_normalize_roles helper that uppercases role/application identifiers and applied it to grant_schema_read_specific, grant_schema_procedure_usage_specific, grant_schema_operate_specific, grant_schema_monitor_specific, grant_database_usage, grant_operate_to_application, grant_usage_to_application, grant_object_application, and grants_smoke_test.
  • grant_schema_object_privileges normalizes roles passed as string or list to uppercase via _grants_normalize_roles, and uses these normalized values when querying and comparing grantees.
  • grant_database_usage now uppercases roles and shares for comparison, and stores existing_roles/existing_shares in uppercase to align with normalized inputs.
macros/grants/_helpers.sql
macros/grants/grant_schema_read.sql
macros/grants/grant_procedure_usage.sql
macros/grants/grant_schema_operate.sql
macros/grants/grant_schema_monitor.sql
macros/grants/grant_schema_object_privileges.sql
macros/grants/grant_database_usage.sql
macros/grants/grant_operate_to_application.sql
macros/grants/grant_usage_to_application.sql
macros/grants/grant_object_application.sql
macros/grants/grants_smoke_test.sql
The grant_object macro is now strictly grant-only, removing all revoke logic and focusing on granting missing privileges to specified roles.
  • Removed OWNERSHIP and revokable_read_privs handling plus all revoke_statements; the macro now only inspects existing grants to avoid duplicate GRANTs.
  • Normalized grant_roles via _grants_normalize_roles, keyed existing_role_priv_map by uppercased grantee_name, and generated only GRANT statements for privileges not yet present.
  • Updated summary logging and README/dbt_project.yml docs to reflect grant-only behavior and version bump to 1.0.7.
macros/grants/grant_object.sql
README.md
dbt_project.yml
New helper macros for bulk grant state inspection and dedicated tests validate helper behavior and idempotency/skip-logic across grant macros.
  • Added _grants_get_schema_object_privs to aggregate privilege_type/grantee pairs from information_schema.object_privileges into a {role: [privs]} map with optional filters.
  • Added _grants_get_schema_grants to collect schema-level privileges (e.g., USAGE) for roles, and _grants_get_future_grants to summarize SHOW FUTURE GRANTS into a {role: [priv:object_type]} map.
  • Introduced test_grants_helpers (13 unit-like Jinja tests) covering _grants_normalize_roles, _grants_format_list, _grants_append_unique, and case-insensitive role membership.
  • Introduced test_grants_idempotency integration macro that uses the new helpers and grant macros against live Snowflake to assert return types, idempotent behavior, and case-insensitive role normalization.
  • Updated grants_smoke_test to exercise mixed-case roles, grant_schema_object_privileges, grant_schema_procedure_usage_specific, grant_object (empty objects), grant_share_read, and grant_database_usage in dry-run mode.
macros/grants/_helpers.sql
integration_tests/macros/test_grants_helpers.sql
integration_tests/macros/test_grants_idempotency.sql
macros/grants/grants_smoke_test.sql
CHANGELOG.md

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 4 issues, and left some high level feedback:

  • In _grants_get_schema_object_privs, the _role variable is constructed but never used; consider removing it or using it to make the intent clearer and avoid confusion.
  • The grant_schema_object_privileges macro’s revoke_query filters only on object_schema and privilege_type but not on object_type, which may cause revokes on unintended object types; consider narrowing the query to the specific object_type being processed.
  • The schemas_skipped tracking in grant_schema_read_specific uses a list with a dummy initial element and length - 1 arithmetic; replacing this with a simple integer counter would make the skip-count logic easier to read and maintain.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- In `_grants_get_schema_object_privs`, the `_role` variable is constructed but never used; consider removing it or using it to make the intent clearer and avoid confusion.
- The `grant_schema_object_privileges` macro’s revoke_query filters only on `object_schema` and `privilege_type` but not on `object_type`, which may cause revokes on unintended object types; consider narrowing the query to the specific `object_type` being processed.
- The `schemas_skipped` tracking in `grant_schema_read_specific` uses a list with a dummy initial element and `length - 1` arithmetic; replacing this with a simple integer counter would make the skip-count logic easier to read and maintain.

## Individual Comments

### Comment 1
<location path="macros/grants/grant_schema_read.sql" line_range="61-70" />
<code_context>
+    {% set existing_privs = dbt_dataengineers_utils._grants_get_schema_object_privs(schema, ['SELECT', 'REFERENCES', 'REBUILD', 'READ'], grant_roles) %}
</code_context>
<issue_to_address>
**issue (bug_risk):** Existing object privileges are checked per-role only, which may skip granting SELECT on some object types.

`_grants_get_schema_object_privs` aggregates privileges per grantee without distinguishing object types or whether they apply to all objects. This code treats any existing `SELECT` for a role as meaning no further `SELECT` grants are needed, and then skips granting on views, materialized views, tables, external tables, dynamic tables, and streams.

As a result, roles that currently have `SELECT` on only some objects may never receive `SELECT` on the remaining ones, changing behavior from the previous `grant ... on all ...` pattern. If you want to maintain full read coverage while keeping the operation idempotent, consider tracking existing privileges per object type or reverting to unconditional `grant ... on all ...` and relying on Snowflake’s idempotence.
</issue_to_address>

### Comment 2
<location path="macros/grants/grant_schema_object_privileges.sql" line_range="58-67" />
<code_context>
+    {% set existing_privs = dbt_dataengineers_utils._grants_get_schema_object_privs(schema_name, permission_list, role_list) %}
</code_context>
<issue_to_address>
**issue (bug_risk):** The bulk helper ignores `object_type`, so revokes/grants may consider privileges on unrelated objects in the schema.

In `grant_schema_object_privileges`, the bulk path calls `_grants_get_schema_object_privs(schema_name, permission_list, role_list)` and builds a revoke query on `information_schema.object_privileges` filtered only by `object_schema` and `privilege_type`, ignoring the `object_type` argument.

As a result:
- A call with `object_type='TABLE'` may treat privileges on views, streams, etc. as if they applied to tables when deciding which roles already have a privilege.
- The revoke logic may target roles based on privileges they hold on other object types, even though the generated statement is `REVOKE ... ON ALL <object_type>s IN SCHEMA ...`.

To avoid incorrect grant/revoke behavior, the queries should either filter by `object_type` or the bulk helper should be limited to cases where its semantics match the object-type-specific logic.
</issue_to_address>

### Comment 3
<location path="macros/grants/_helpers.sql" line_range="125-134" />
<code_context>
+{% macro _grants_get_schema_object_privs(schema, privilege_types, grantees) %}
</code_context>
<issue_to_address>
**suggestion (bug_risk):** Unused `_role` variable and mixed casing in the grantee filter can be cleaned up for clarity.

Inside `_grants_get_schema_object_privs`:
- `_role = row[0] ~ '::' ~ row[1]` is computed but never used.
- `priv_filter` is uppercased, but `grantees` is not, even though callers currently pass uppercased roles via `_grants_normalize_roles`.

Please remove `_role`, and either normalize `grantees` here or clearly document/enforce the expectation that callers pass uppercased values (e.g. via a `grantees_upper` parameter name) to avoid subtle casing-related bugs.

Suggested implementation:

```
{# Bulk check: returns dict {role: [privs]} for a given schema from information_schema.object_privileges.
   Filters by optional privilege_types list and grantee list. Grantees are normalized to UPPER() to avoid casing issues. #}
{% macro _grants_get_schema_object_privs(schema, privilege_types, grantees) %}
    {% set result_map = {} %}
    {% set priv_filter = privilege_types | map('upper') | list %}
    {% set grantees_upper = grantees | map('upper') | list %}
    {% set query %}
        select privilege_type, grantee
        from information_schema.object_privileges
        where object_schema = '{{ schema }}'
        {% if priv_filter | length > 0 %}
          and privilege_type in ({{ priv_filter | map('tojson') | join(', ') }})
        {% endif %}
        {% if grantees_upper | length > 0 %}

```

1. Remove the unused `_role = row[0] ~ '::' ~ row[1]` assignment inside `_grants_get_schema_object_privs`; the local `_role` variable is not referenced anywhere and should simply be deleted.
2. Anywhere in `_grants_get_schema_object_privs` that currently uses `grantees` to build the SQL (e.g. an `and grantee in (...)` clause) or to filter results in Python/Jinja should be updated to use `grantees_upper` instead:
   - In the SQL, compare against `UPPER(grantee)` or ensure grantees are uppercased consistently with `grantees_upper`.
   - In any subsequent logic that checks membership or builds `result_map`, use `grantees_upper` for comparison to ensure casing-safe behavior.
3. If there is helper documentation or comments elsewhere that mention that callers must pass uppercased grantees (e.g. via `_grants_normalize_roles`), update those comments to reflect that `_grants_get_schema_object_privs` now performs its own uppercasing internally.
</issue_to_address>

### Comment 4
<location path="macros/grants/grant_schema_operate.sql" line_range="21-30" />
<code_context>
+{% macro _grants_get_schema_object_privs(schema, privilege_types, grantees) %}
+    {% set result_map = {} %}
+    {% set priv_filter = privilege_types | map('upper') | list %}
+    {% set query %}
+        select privilege_type, grantee
+        from information_schema.object_privileges
</code_context>
<issue_to_address>
**question (bug_risk):** Revoke statements now use bulk `revoke operate on all tasks/pipes` which may be broader than the previous per-object revokes.

Previously, revokes were per object:
```sql
revoke operate on <object_type> in schema <db>.<object> from role <role>;
```
Now, for non-managed roles, you use bulk revokes:
```sql
revoke operate on all tasks in schema ... from role ...;
revoke operate on all pipes in schema ... from role ...;
```
This changes semantics: a role that had OPERATE on only some tasks/pipes will now lose it on *all* tasks/pipes in the schema, including grants not originally managed by this macro.

Please confirm whether the macro is intended to fully own OPERATE on tasks/pipes for the schema. If not, consider reverting to per-object revokes or scoping bulk revokes to objects discovered by a dedicated query to avoid unintentionally removing other grants.
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

Comment thread macros/grants/grant_schema_read.sql Outdated
Comment thread macros/grants/grant_schema_object_privileges.sql Outdated
Comment thread macros/grants/_helpers.sql Outdated
Comment on lines 21 to 30
{% set query %}
select object_type, concat(object_schema, '.', object_name) as object_name, privilege_type, grantee
select privilege_type, grantee
from information_schema.object_privileges
where privilege_type = 'OPERATE' and object_schema = '{{ schema }}'
{% endset %}
{% set results = run_query(query) %}
{% if execute and results %}
{% for row in results %}
{% set priv = row[2] %}{% set grantee = row[3] %}
{% set priv = row[0] %}{% set grantee = row[1] %}
{% if priv == 'OPERATE' %}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question (bug_risk): Revoke statements now use bulk revoke operate on all tasks/pipes which may be broader than the previous per-object revokes.

Previously, revokes were per object:

revoke operate on <object_type> in schema <db>.<object> from role <role>;

Now, for non-managed roles, you use bulk revokes:

revoke operate on all tasks in schema ... from role ...;
revoke operate on all pipes in schema ... from role ...;

This changes semantics: a role that had OPERATE on only some tasks/pipes will now lose it on all tasks/pipes in the schema, including grants not originally managed by this macro.

Please confirm whether the macro is intended to fully own OPERATE on tasks/pipes for the schema. If not, consider reverting to per-object revokes or scoping bulk revokes to objects discovered by a dedicated query to avoid unintentionally removing other grants.

…idempotency

- _grants_get_schema_object_privs: removed unused _role variable, added
  internal grantees uppercase normalization, added optional object_type
  parameter to filter by specific object type
- grant_schema_read_specific: removed incorrect per-privilege skip logic
  that aggregated across object types. GRANT ON ALL is idempotent in
  Snowflake so always issue it. Only skip schema USAGE (single check)
  and future grants (which error on duplicates)
- grant_schema_object_privileges: now passes object_type to both the
  helper query and the revoke query to avoid cross-object-type leakage

.... Generated with [Cortex Code](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code)

Co-Authored-By: Cortex Code <noreply@snowflake.com>
@jonhopper-dataengineers jonhopper-dataengineers merged commit 9a6f3f6 into main Jun 25, 2026
2 checks passed
@jonhopper-dataengineers jonhopper-dataengineers deleted the feature/grant-object-remove-revokes branch June 25, 2026 08:33
jonhopper-dataengineers added a commit that referenced this pull request Jun 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant